The background description provided herein is for the purpose of generally presenting the context of the disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
In computing, telerobotics is an area of robotics concerned with the control of semi-autonomous robots from a distance. As used herein, a “robot” is a computer-controlled machine controlled relatively indirectly, such as by being programmed. As used herein, a “telerobot” refers to a computer controlled machine which is controlled remotely by a human operator. Another term for “telerobot” is “telemanipulator”.
Remote human control can involve direct correspondence between a human operator's command actions and the actions of the telerobot, such as a radio-controlled model aircraft in which the engine, flaps, and rudder are controlled directly by a human operator in real time. Remote human control can also be indirect, such as when the telerobot comprises logic to control engine, flaps, and rudder and the human controller's input is interpreted as a change in speed and/or direction and a rate at which the telerobot will implement the change in speed and/or direction. The human operator's command actions can be communicated in essentially real time, that is to say, with a latency of less than one-quarter of a second, or not in real time (with a latency of greater than one-quarter second).
As used herein, an “avatar” is a graphical representation of a person, animal, or physical object in a virtual world, such as a computer game or computer generated 3-dimensional space, wherein the graphical representation is controlled by a living human, animal, or other organism (collectively referred to herein as an “operator” or “human operator”).
When an operator is controlling a telerobot in the real world or an avatar in a digital space, feedback from the telerobot or avatar is typically available or provided to the operator. Feedback may be through direct observation by the operator (as may be the case with a radio-controlled aircraft) or by transmitting sensor data from the telerobot to the operator. Examples of sensor data include video (images and audio), attitude (roll, pitch, and yaw), speed, and the like. The sensor data may be decoded or otherwise converted into output which the operator can perceive. For example, decoded (or analog) video and decoded attitude data may be output to a video on a screen viewable by the operator, attitude data may be decoded and output to a graphical representation of attitude indicators on a video screen or to a joystick held by the human operator as haptic feedback. In the case of an avatar, the feedback may be provided in similar ways, though with a first step of generating the sensor data, such as generating video from a point of view within a computer generated 3-dimensional space.
The feedback provided to the operator may be “immersive”, in that it replaces some or all of the sensory experiences of the operator, such as with a virtual reality headset which displays images (including video), provides stereo audio, and/or haptic feedback derived from data obtained by sensors in, available, or proximate to the telerobot; this shall be referred to herein as “immersive feedback”.
With respect to both telerobots and avatars, the operator's control commands may be interpreted from real world movements of the operator or even from detecting activity of the operator's nervous system. In the simple case of the radio-controlled aircraft, a human operator may control one or two joy sticks, wheels, pedals, buttons, or the like in a controller. The human operator's movements of these components in the controller produces electrical signals which are (typically processed and) then transmitted to the remote aircraft. An operator's movements may also be observed—for example, by a video camera, by movement of a sensor-bearing “wand” or the like—interpreted as control commands, encoded, and then transmitted to the remote telerobot or avatar. This shall be referred to herein as “movement mapping based control”.
When movement mapping based control is provided between an operator and a telerobot or avatar, and when the telerobot or avatar provides immersive auditory, visual, or haptic feedback to the operator (“immersive feedback”), problems can occur when there is a difference between the real space occupied by the operator and the real or virtual space occupied by the telerobot or avatar. For example, if the operator is wearing a virtual reality headset displaying immersive feedback, such as video recorded by a telerobot, and if movement mapping based control is used to control the telerobot, the immersive feedback may cause the operator to experience dizziness or confusion or the operator may not be able to properly control the telerobot if the real space navigated by the operator and the telerobot do not align.